Electronic networks such as the Internet have become a pervasive way of transferring not only communications information, but critical personal, company and even entertainment data. As more and more companies maintain their critical information in electronic format, users face the need to transfer increasingly large files (>100 GB) from one user site to another; for example video data, CAD drawing data, scientific instrument data, and other datasets as will be apparent to the reader. While many electronic network backbones have adequate bandwidth to handle large data transfers, the users' access links to these backbones typically have limited bandwidth (e.g., T1 at 1.5 Mbps or T3 at 44.7 Mbps). Consider, for example, transferring a 100 GB file over such links—with a T1 access capability it would take more than 6 days and even with T3 access over 5 hours. Even those times assume the access links are only used for these file transfers and not otherwise utilized.
Both network service providers and the customers of network service providers are facing the problem of how to effectively handle increasingly large data volume and data size transfers. The different parties have different solutions to this problem. The incumbent local exchange carriers (ILECs) that own the access links to the customer premises typically sell the customers more access links. End-users and system integrators sometimes use end-point solutions that virtually increase the bandwidth of the network with caching and compression techniques before data is sent to the wide area network (WAN). For example, WAN accelerators/optimizers such as the PeriScope™ product from Peribit Networks perform compression before user data is passed to the network and/or after it has been received from the network. Other WAN accelerator vendors include Network Executive Software Inc. (NetEx) and Swan Labs Corp.
Another type of virtual bandwidth improvement product, a Wide Area File Service (WAFS), accelerates remote file access through a WAN to a data center, whereby to appear as fast as a LAN file system access, using caching and other techniques. WAFS vendors include Cisco, DiskSites Inc., FineGround Networks, Riverbed Technology Inc., and Tacit Networks Inc.
Yet another type of virtual accelerator, CDNs (Content Distribution Networks) are used to speed up the delivery of web pages to large numbers of receivers (readers). The concern typically addressed by a CDN is not with the size of data to be transferred, but the number of readers it is to be distributed to. Thus, CDNs are designed to improve server bottlenecks.
While providing the appearance of temporary improvements in limited areas, the above techniques do not address the fundamental problem of data transfer sizes increasing faster than end-to-end network bandwidths are capable of timely handling.
In summary, network user data size and transfer requirements are exceeding the capability of existing network infrastructures and services to effectively handle. While limited acceleration solutions exist for portions, typically end-portions, of the network, these are not comprehensive or long-term solutions. Given the fundamental constraints of the existing network infrastructure, the present inventors have recognized the need for a more comprehensive and long-term solution to large, timely file transfer requirements.